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Abstract 

The  objective  of  this  report  is  to  describe  the  methods  used  in  and  lessons  learned 
from  a  research  study  that  evaluated  prescriptions  for  medication  errors  prior  to 
the  implementation  of  an  ambulatory  computerized  prescriber  order  entry 
(ACPOE)  system  in  a  community-based,  integrated  health  system.  Several  aspects 
of  the  study  are  described:  practice  setting,  research  team,  project  scope  and  study 
design,  data  elements,  database  creation,  data  sources,  study  methods,  evaluation 
methods,  refinements,  and  current  project  status.  Lessons  learned  are  summarized. 
Focused  attention  to  these  aspects,  a  priori,  resulted  in  the  collection  of  data  that 
have  been  used  to  characterize  the  epidemiology  of  medication  errors  at  baseline. 
By  repeating  these  same  methods  post- ACPOE  implementation,  the  impact  of  the 
ACPOE  system  on  medication  errors  will  be  determined.  The  methods  used  to 
measure  the  baseline  rate  of  medication  errors  provided  useful  results  and  proved 
practical  in  a  practitioner-based  organization.  This  design  can  be  replicated  in 
similar  organizations  desiring  to  evaluate  the  impact  of  an  ACPOE  system  on 
medication  errors. 


Introduction 

During  the  millennium  period,  the  issue  of  patient  safety  was  elevated  to 
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national  focus,  based  largely  on  the  publications  of  the  Institute  of  Medicine. 

To  proactively  address  these  patient  safety  concerns,  in  2002,  the  leadership  of  the 
Everett  Clinic,  Snohomish  County,  Washington,  began  adding  an  ambulatory 
computerized  prescriber  order  entry  (ACPOE)  system  to  their  existing 
computerized  medical  record  (CMR).  As  a  priority,  the  clinic  leadership  initiated 
this  study  to  evaluate  the  impact  of  the  ACPOE  system  on  patient  safety.  The 
purpose  of  this  study  was  twofold:  (1)  to  measure  the  baseline  incidence  of 
medication  errors  prior  to  implementation  of  an  ACPOE  system,  and  (2)  to 
characterize  the  epidemiology  of  these  errors.  This  report  describes  the  methods 
used  to  conduct  the  preimplementation  evaluation  in  one  internal  medicine  clinic 
selected  as  the  pilot  site.  Addressed  are  the  developments  of  research  methods; 
decisions  made  to  streamline  the  methods;  and  lessons  learned  in  addressing  the 
challenges  inherent  in  measuring  the  baseline  rate  of  medication  errors  in  a 
practitioner-based,  ambulatory  care  setting. 
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Practice  setting 

The  Everett  Clinic  is  an  integrated,  multispecialty,  physician  group  practice 
that  provides  comprehensive  ambulatory  care  services  for  275,000  residents  in  the 
northern  Puget  Sound  region  of  Washington  State.  The  clinic  espouses  three  core 
values:  (1)  We  do  what  is  right  for  each  patient;  (2)  we  provide  an  enriching  and 
supportive  work  environment;  and  (3)  our  team  focuses  on  value,  service,  quality, 
and  cost.  An  integral  part  of  this  value  system  is  the  promotion  of  a  culture  of 
safety. 

The  clinic  is  both  owned  and  managed  by  local  physicians  in  the  community. 
The  clinic  health  care  team  includes  225  physicians  working  collectively  with  600 
other  health  care  professionals.  Care  is  delivered  in  13  locations,  in  60  clinics, 
throughout  Snohomish  County,  Washington.  The  care  delivery  system  includes 
primary  care  and  specialty  clinics,  walk-in  clinics,  two  outpatient  surgery  centers, 
a  cancer  center,  comprehensive  laboratory,  and  imaging  services.  A  hospitalist 
team  functions  24  hours  daily,  7  days  weekly,  and  admits  to  the  local  community 
hospital.  The  clinic  physicians  write  more  than  2.1  million  prescriptions  each 
year,  amounting  to  approximately  $136  million  dollars.  Ten  percent  of  these 
prescriptions  are  filled  at  three  onsite  pharmacies  owned  and  operated  by  the 
clinic.  The  clinic  has  estimated  the  ACPOE  system  will  save  $1.8  million  dollars 
annually  in  prescription  processing  costs  (2002  data). 

The  clinic  has  a  strong  medication  management  team  that  manages  the 
formularies  of  the  12  health  plans  the  clinic  contracts  with.  Two  clinical 
pharmacists  (JWN  and  NML)  optimize  the  quality  of  medication  use  and  provide 
cost  management  by  collaboratively  working  with  the  physicians.4  Together,  the 
physicians  and  pharmacists  lead  the  Pharmacy  and  Therapeutics  Committee  and 
are  members  of  the  Quality  Improvement  Committee.  These  two  committees 
oversee  the  patient  safety  aspects  of  the  ACPOE  implementation. 

The  information  technology  (IT)  infrastructure  at  the  clinic  is  provided  by 
employees  of  a  wholly-owned  subsidiary,  Clinitech®  Information  Services. 
Clinitech  employees  have  worked  collaboratively  with  the  clinical  and 
administrative  staff  since  1995  to  develop  the  clinic’s  CMR.5  At  present,  the 
CMR  incorporates  physician  dictation  (chart  notes  summarizing  each  ambulatory 
care  visit),  laboratory  values,  and  radiology  reports.  The  ACPOE  system  is  the 
next  module  being  implemented.  The  CMR  also  serves  as  the  backbone  for  future 
development  of  clinical  decision  support  (CDS)  tools  for  clinicians  to  use 
throughout  the  health  care  system.  These  CDS  tools  will  largely  be  presented  as 
programmed  enhancements  of  the  ACPOE  system.  The  Clinitech  team  has  also 
developed  and  maintained  an  extensive  intranet  for  physician  and  staff  use,  which 
helps  the  latter  easily  access  clinical  and  administrative  information  geared  to 
helping  them  provide  patient  care. 

The  ACPOE  system  is  Web-based  and  involves  point  and  click  functionality. 

It  integrates  patient  scheduling,  chart  notes,  laboratory  values,  radiology,  and 
prescribing  into  one  system.  The  ACPOE  software  module  was  built  by  the 
Clinitech  team  de  novo,  not  purchased  from  an  outside  vendor.  A  discussion  of 
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software  development  and  the  selection  of  portable,  wireless  order  entry  devices 
will  be  provided  in  a  separate  publication.  When  a  prescription  is  written 
electronically,  it  is  saved  on  the  clinic  mainframe  computer  and  can  be  printed 
and  handed  to  the  patient,  or  it  can  be  automatically  faxed  to  a  retail  phannacy. 
The  clinic  currently  auto  faxes  to  more  than  100  pharmacies.  The  prescribing 
software  does  not  currently  allow  for  full  electronic  transfer  to  retail  pharmacy 
order  entry  software.  Therefore,  the  printed  prescription  is  not  eliminated,  and  the 
retail  pharmacy  maintains  this  printed  order  as  part  of  the  patient  record. 


Conducting  the  study 

Establishing  the  research  team 

The  clinic  leadership  partnered  with  investigators  from  the  University  of 
Washington  Phannaceutical  Outcomes  Research  and  Policy  Program  to  evaluate 
the  impact  of  the  ACPOE  system  on  medication  errors.  From  mid-2002  through 
early  2003,  this  research  team  developed  the  plan  and  conducted  the  research  that 
evaluated  medication  errors  in  one  internal  medicine  clinic,  prior  to  implementing 
the  ACPOE  system  in  July  2003.  The  team  consisted  of  four  core  investigators 
(JWN,  NML,  EBD,  TKH),  three  clinical  pharmacists  who  served  as  research 
associates  in  evaluating  the  prescriptions  for  errors  (KK,  ST,  CW),  and  a  database 
programmer  who  was  also  a  clinically  trained  pharmacist  (RH).  One  of  the 
research  associates  was  a  specialist  in  geriatrics  (KK)  and  one  in  primary  care 
(CW).  Physician  leaders  and  administrators  of  the  clinic  provided  oversight.  For  7 
months  the  team  met  weekly,  in  person.  All  decisions  related  to  study  design  were 
captured  in  written  meeting  minutes  and  in  an  electronic  e-mail  file.  When  the 
study  was  launched,  communication  took  place  almost  daily.  At  that  juncture, 
efficiencies  were  gained  by  managing  communications  via  teleconferences;  e- 
mail  was  also  extensively  used. 

Defining  the  scope  and  study  design 

The  first  step  in  defining  the  scope  of  the  study  was  to  diagram  the 
prescription- flow  process  in  the  ambulatory  care  setting  at  the  clinic  (Figure  1). 
This  exercise  validated  the  notion  that  the  process  of  prescribing  in  the 
ambulatory  care  setting  is  complex,  and  that  there  are  numerous  points  in  that 
process  where  errors  can  occur. 

The  second  step  was  to  decide  whether  to  adopt  a  concurrent  or  retrospective 
method  for  data  collection.  The  team  considered  using  a  concurrent,  observational 
study  design,  following  prescriptions  as  they  made  their  way  through  the  clinic, 
from  the  point  of  prescriber  writing  to  the  point  of  dispensing  at  the  pharmacy. 
Barker  and  Flynn  have  recommended  this  approach  for  conducting  drug 
dispensing  and  administration  studies.  ’  However,  the  goals  of  the  present  study 
were  to  identify  errors  occurring  solely  during  the  prescribing  process,  which 


187 


Advances  in  Patient  Safety:  Vol.  2 


Figure  1.  New  Prescription-Handwritten  Process 


could  be  measured  more  efficiently  by  using  a  retrospective  design.  This  provided 
a  method  for  identification  and  characterization  of  errors,  albeit  without  observing 
the  specific  point  in  the  prescription  process-flow  where  each  error  occurred. 

The  third  step  was  to  decide  which  pharmacies  to  include  in  the  study.  Using 
pharmacies  onsite  at  the  clinic  would  facilitate  capture  of  errors  inclusive  of  the 
point  at  which  they  were  entered  into  the  pharmacy  computer  system.  Using 
phannacies  external  to  the  clinic  would  preclude  information  obtained  at  this 
point.  As  the  potential  exists  for  electronically  written  prescriptions  to  be 
electronically  transmitted  directly  into  the  phannacy  computer  system 
(eliminating  the  faxing  step),  it  seemed  prudent  to  include  this  point  in  the 
measurement  process.  Thus,  study  methods  included  prescriptions  transmitted  to 
and  filled  solely  at  a  pharmacy  owned  and  operated  by  the  clinic.  The 
identification  and  characterization  of  medication  errors  occurring  in  the 
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dispensing  process  was  outside  the  scope  of  this  study,  as  the  ACPOE  system 
would  not  impact  this  activity. 

The  fourth  step  was  to  decide  which  clinic  should  be  first  brought  online  with 
the  ACPOE  system.  An  internal  medicine  clinic,  where  prescribers  were  eager  to 
prescribe  electronically,  was  selected  as  the  pilot  site.  The  research  team  reasoned 
that  selecting  this  clinic  would  maximize  the  chances  of  successful 
implementation  not  only  within  the  pilot  clinic,  but  that  the  spread  of  this  positive 
news  would  facilitate  implementation  at  the  additional  59  clinics  that  comprise 
the  Everett  Clinic  integrated  health  system. 

Defining  and  establishing  data  elements 

The  research  team  conducted  a  comprehensive  review  of  the  literature'  and 

used  this  information  to  draft  the  data  elements.  Many  of  the  data  elements  used 
by  previous  investigators  were  selected  for  their  use  in  the  inpatient  setting;8  10' 12' 14  16 
not  all  of  them  were  applicable  to  the  ambulatory  setting.  The  team  selected  those 
that  were  applicable,  including  error  characteristics  that  specified  type  of  error 
and  underlying  cause  of  error,  as  well  as  patient-,  prescriber-,  and  prescription- 
level  characteristics.  The  availability  of  the  CMR  chart  notes  and  laboratory 
values  made  it  possible  to  evaluate  clinical  errors,  as  well  (Table  1).  Data  were 
recorded  on  every  prescription  evaluated,  regardless  of  whether  an  error  was 
identified. 

To  conduct  a  risk  assessment  on  each  error  identified,  the  definition  of  error 
and  the  Risk  Assessment  Index  published  by  the  National  Coordinating  Council 
on  Medication  Error  Reporting  and  Prevention  (NCC  MERP)17was  adopted. 
According  to  NCC  MERP,  “A  medication  error  is  any  preventable  event  that  may 
cause  or  lead  to  inappropriate  medication  use  or  patient  harm  while  the 
medication  is  in  the  control  of  the  health  care  professional,  patient,  or  consumer. 
Such  events  may  be  related  to  professional  practice,  health  care  products, 
procedures,  and  systems,  including  prescribing;  order  communication;  product 
labeling,  packaging,  and  nomenclature;  compounding;  dispensing;  distribution; 
administration;  education;  monitoring;  and  use.”  We  purposefully  limited  the 
scope  of  our  study  to  include  errors  related  solely  to  prescribing.  The  NCC  MERP 
severity  levels  range  from  “A”  (circumstances  that  have  the  capacity  to  cause 
error),  to  “I”  (an  error  occurred  that  resulted  in  patient  death)  (Table  2). 

Variation  among  research  associates  in  identifying  errors  is  inherent  in  the 
methodologic  process  chosen.  To  prevent  this,  a  concerted  effort  was  made  to 
codify  definitions  and  rules,  to  ensure  a  standardized  approach  to  the 
identification  of  errors  and  interpretation  of  data  reviewed  by  the  research 
associates.  For  example,  a  standardized  definition  was  needed  for  the  data 
element  “information  missing  from  prescription.”  In  the  clinic  setting,  when  a 
prescriber  orders  a  medication  to  be  taken  orally,  the  abbreviation  “PO”  (per  os, 
by  mouth)  is  seldom  written  on  the  prescription.  Thus,  the  team  established  a  rule 
that  prescriptions  intended  for  oral  consumption,  from  which  a  route  of 
administration  was  absent,  did  not  constitute  an  error.  In  contrast,  prescriptions 
that  were  intended  for  other  than  oral  administration,  that  did  not  specify  the  route 
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Table  1.  Data  elements  and  error  characteristics 


Error  characteristics — type  of  error 

Illegible  prescription 

Information  missing 

Inappropriate  abbreviations  (drug  name,  apothecary  system,  leading/trailing  zeros,  use  of  “u” 
for  units) 

Wrong  information  (wrong  patient,  wrong  drug,  wrong  route,  wrong  dosage  form,  wrong  dose, 
wrong  strength,  wrong  directions) 

Clinical  criteria  -  patient  allergy 

Clinical  criteria  -  drug-drug  interaction 

Clinical  criteria  -  drug-disease  interaction 

Clinical  criteria  -  therapeutic  duplication 

Clinical  criteria  -  Contraindication  -  pregnancy,  geriatric  criteria,  Laboratory  Harm  Table 

Clinical  criteria  -  multiple  prescribers  prescribing  the  same  drug  or  drug  class 

Error  characteristics — underlying  cause  of  error 

Drug  name  confusion 

Lack  of  standardization  of  prescription  processing  rules 

Lack  of  knowledge  of  drug  or  lack  of  drug  information  by/for  prescriber 

Lack  of  information  about  the  specific  patient  (missing  lab  values,  disease  states,  interactions) 

Staffing  problems 

Unable  to  determine 

Patient  level 

Patient  year  of  birth 

Patient  gender 

Prescriber  level 

Prescriber  specialty 

Prescriber  year  of  birth 

Prescriber  years  in  practice 

Prescription  level 

Method  of  prescription  transmittal 

Error  detected  and  corrected  during  the  routine  course  of  the  prescribing  process 

Formulary-therapeutic  interchange  required 

Formulary-prior  authorization  required 

Prescription  data  elements 

Name,  dose  and  directions  for  prescription  (both  as  written  and  as  should  have  been  written) 
Therapeutic  drug  class 

NCC  MERP  Risk  Assessment  Index  -  Severity  Levels  A  through  1 

NCC  MERP  =  National  Coordinating  Council  on  Medical  Error  Reporting  and  Prevention 
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Table  2.  NCC  MERP  Definition  of  a  medication  error  and  Risk  Assessment  Index17 


Category 

Description  of  category 

No  error 

A 

Circumstances  or  events  that  have  the  capacity  to  cause  error 

Error,  no  harm 

B 

An  error  occurred,  but  the  medication  did  not  reach  the  patient 

C 

An  error  occurred  that  reached  the  patient  but  did  not  cause  patient 
harm 

D 

An  error  occurred  that  resulted  in  the  need  for  increased  patient 
monitoring  but  no  patient  harm 

Error,  harm 

E 

An  error  occurred  that  resulted  in  the  need  for  treatment  or 
intervention  and  caused  temporary  patient  harm 

F 

An  error  occurred  that  resulted  in  initial  or  prolonged  hospitalization 
and  caused  temporary  patient  harm 

G 

An  error  occurred  that  resulted  in  permanent  patient  harm 

H 

An  error  occurred  that  resulted  in  a  near-death  event 

(e.g.,  anaphylaxis,  cardiac  arrest) 

Error,  death 

1 

An  error  occurred  that  resulted  in  patient  death 

NCC  MERP  =  National  Coordinating  Council  on  Medical  Error  Reporting  and  Prevention 


of  administration,  did  constitute  an  error.  An  example  of  the  latter  was  a 
prescription  written  for  “Rhinocort,  2  sprays  each  side,  bid”  (bis  in  d’ie,  twice 
daily).  The  team  considered  this  prescription  to  contain  an  error,  as  the  route  of 
administration,  “intranasally,”  was  not  included  on  the  prescription.  This  decision 
engendered  lively  debate  within  the  team.  It  was  argued  that  a  health  care 
professional  would  likely  know  the  route  of  administration,  whereas  the  patient 
would  not.  After  much  discussion  the  rule  described  above  was  codified. 
Separately,  the  team  created  a  list  of  “inappropriate  abbreviations.”  It  included 
drug  name  abbreviations,  the  use  of  the  terminology  from  the  apothecary  system, 
the  use  of  “u”  for  the  word  “units,”  and  the  lack  of  leading  or  the  presence  of 
trailing  zeros. 

The  research  team  next  established  rules  for  clinical  errors  that  constituted 
“contraindications.”  Specifically,  the  team  was  very  interested  in  identifying 
errors  occurring  in  patients  greater  than  65  years  of  age.  The  use  of  Beers 
criteria  — a  list  of  medications  that  should  be  avoided  in  the  elderly — was 
adopted  for  this  purpose.  The  team  also  established  rules  for  what  constituted 
appropriate  laboratory  monitoring  in  patients  of  all  ages.  To  achieve  this  end,  a 
Laboratory  Harm  Table,  specific  to  the  clinic’s  population  and  prescribing 
patterns,  was  created.  The  Laboratory  Hann  Table  specified  laboratory¬ 
monitoring  parameters  that,  if  not  ordered  on  a  patient-specific  basis,  could  result 
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in  patient  harm,  and  consequently  be  considered  a  medication  error.  In  creating 
this  evaluation  tool,  the  team  was  careful  to  differentiate  laboratory  values  that 
would  optimize  medication  effectiveness  from  those  that  are  necessary  for 
preventing  harm;  only  the  latter  were  included.  For  example,  ordering  a  baseline 
set  of  liver  function  tests  prior  to  instituting  therapy  with  hydroxymethyl-glutaryl 
coenzyme  A  (HMG-CoA)  reductase  inhibitor  (statin)  was  included  in  the 
Laboratory  Harm  Table,  whereas  the  ongoing  monitoring  of  a  lipid  panel  to  assess 
statin  efficacy  was  not.  A  copy  of  the  Laboratory  Harm  Table  is  available  from 
the  investigators.  The  clinic  Pharmacy  and  Therapeutics  Committee  approved  the 
use  of  Beers  criteria  and  the  Laboratory  Harm  Table  specifically  for  this  research 
project.  The  interpretation  of  clinical  errors  due  to  drug-disease  interactions  and 
drug-drug  interactions  was  left  to  each  individual  research  associate.  All  had 
recently  completed  their  clinical  training  and  were  employed  in  clinical  practices. 
The  team  reasoned  that  drug-disease  interactions  are  addressed,  in  part,  by  the 
Laboratory  Harm  Table.  In  hindsight,  creating  a  list  of  the  50  most  clinically 
significant  drug-drug  interactions  was  certainly  a  manageable  task  and  could 
easily  have  been  codified. 

Each  rule  was  codified  in  the  Operations  Manual.  The  investigators  also 
created  a  Reference  Packet  for  use  by  the  research  associates.  The  Reference 
Packet  included  an  example  of  the  signature  of  each  prescriber  whose 
prescriptions  were  undergoing  review,  a  copy  of  the  Laboratory  Harm  Table,  the 
NCC  MERP  Risk  Assessment  Index,  and  contact  information  for  the 
investigators. 

Creating  the  study  database 

A  member  of  the  team  who  had  both  clinical  and  IT  expertise  created  the 
database  (RH).  The  database  was  created  in  a  popular  relational  database 
program,  with  five  screens  that  facilitated  entry  of  data  elements,  utilizing  a 
design  schema  that  was  clinically  logical  and  relevant.  Important  features  of  the 
database  included  the  automatic  downloading  of  eligible  prescriptions  from  the 
pharmacy  computer  system  and  numerous  background  tables,  including 
therapeutic  drug  class  tables  and  contraindication  tables.  Automatic  downloading 
of  information  from  the  pharmacy  computer  system  made  useful  information 
immediately  accessible,  without  the  research  associates  having  to  “retrieve”  it 
from  the  CMR.  The  background  tables  included  those  that  linked  drugs  to  dosage 
forms  and  doses,  provided  directions  for  use  (“sigs”),  provided  diagnosis  codes 
based  on  the  International  Classification  of  Diseases  9th  Revision  Clinical 
Modification  system  (ICD-9-CM),19  and  provided  documented  allergies. 

Data  sources 

Three  data  sources  were  utilized  for  the  evaluation  of  medication  errors:  the 
prescription  itself,  the  chart  notes  and  laboratory  values  of  the  CMR  for  clinical 
data,  and  the  prescription  as  entered  into  the  pharmacy  computer  system  onsite. 
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Study  methods 

Approval  for  the  study  was  obtained  from  the  Human  Subjects  Review 
Committee  of  the  University  of  Washington,  using  a  waiver  of  consent.  The 
sample  size  was  calculated  based  on  infonnation  from  the  current  literature,  using 
a  baseline  error  rate  of  5  percent.9  The  study  was  powered  to  provide  an  80 
percent  chance  of  finding  a  40  percent  reduction  in  the  rate  of  medication  errors 
when  comparing  pre-  to  postimplementation  (two-sided  test  of  proportions  with 
an  alpha  level  of  0.05).  Thus  1,500  prescriptions  were  evaluated  in  the 
preimplementation  phase.  The  same  number  will  be  evaluated  for  the 
postimplementation  phase. 

The  method  by  which  prescriptions  were  selected  for  evaluation  was  as 
follows.  A  5-month  time  frame  was  identified;  in  an  effort  to  minimize  bias,  one 
that  occurred  immediately  prior  to  the  time  when  implementing  an  ACPOE 
system  was  first  discussed.  Using  the  beginning  of  this  5-month  time  frame  as  the 
index  date,  every  prescription  written  for  a  patient  in  the  internal  medicine  clinic 
and  filled  at  the  onsite  pharmacy,  from  that  date  forward,  was  evaluated  for 
inclusion  in  the  study.  A  total  of  2,250  prescriptions  were  reviewed  to  find  the 
evaluable  1,500  that  met  the  inclusion  criteria.  The  evaluation  was  limited  to 
newly  written  prescriptions,  inclusive  of  those  written  for  patients  between  the 
ages  of  18  and  89  years.  Prescriptions  transferred  in  from,  or  out  to,  outside 
phannacies  were  excluded.  Other  exclusions  were  prescriptions  for  devices  or 
laboratory  monitoring  equipment  (e.g.,  glucose  test  strips),  prescriptions  filled  via 
medication  vouchers  or  coupons,  and  voided  prescriptions.  The  initial  plan  was 
that  each  prescription  would  be  evaluated  by  two  research  associates  with 
discrepancies  adjudicated  by  the  third;  problem  prescriptions  would  be  evaluated 
by  the  entire  research  team. 

The  team  agreed  at  the  outset  that  chart  notes  in  the  CMR  would  be  evaluated 
for  the  15  months  prior  to  the  writing  of  the  prescription  undergoing  evaluation. 
The  team  reasoned  that  each  patient  should  be  seen  a  minimum  of  once  yearly;  1 5 
months  provided  an  additional  three  months  allowing  for  delays  in  the 
appointment-making  process.  Laboratory  data  were  followed,  retrospectively,  for 
1  year  prior  to  prescription  writing.  The  medication  regimen  of  each  patient  was 
reviewed  in  a  comprehensive  fashion,  evaluating  drug-drug  and  drug-disease 
interactions  for  all  current  medications,  not  limited  to  those  impacting  the 
prescription  undergoing  evaluation.  The  team  also  outlined  a  substudy  that  would 
compare  infonnation  found  in  the  CMR  with  that  found  in  the  pharmacy 
computer  system.  The  intent  was  to  compare,  from  these  two  sources,  the  active 
drug  list,  the  list  of  current  disease  states,  and  allergies. 

Evaluation  methods 

The  research  associates  evaluated  each  prescription  for  medication  enors 
using  a  structured,  computer-based  system.  Using  five  screens,  they  progressed  in 
sequence,  entering  all  data,  one  prescription  at  a  time,  into  the  database  (except 
that  which  was  downloaded  automatically).  The  first  screen  prompted  entry  of 
data  directly  from  the  handwritten  prescription.  The  second  screen  prompted  the 
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evaluation  of  clinical  information  from  the  chart  notes  and  laboratory  values 
contained  in  the  patient’s  CMR.  The  third  screen  prompted  entry  of  data  that 
facilitated  a  comparison  between  the  prescription  as  it  had  been  written  by  the 
prescriber,  and  as  it  had  been  entered  into  the  pharmacy  computer  system  by  the 
dispensing  pharmacist.  Close  attention  was  paid  to  accurate  entry  to  minimize  the 
introduction  of  errors  into  the  study  process.  Once  the  data  were  entered,  the 
fourth  and  fifth  screens  prompted  the  research  associate  to  record  the  detail  of  any 
medication  errors  found.  If  more  than  one  error  was  found  per  prescription,  each 
error  was  evaluated  separately. 

Refinements 

Refinements  in  the  process  took  place  during  the  pretest  phase  of  data  entry — 
data  entry  of  the  first  35  prescriptions.  Several  issues  arose  and  each  was 
addressed  in  turn.  At  the  outset,  each  prescription  required  approximately  45 
minutes  for  data  entry  and  evaluation.  Lack  of  proficiency  with  the  process,  at 
least  initially,  contributed  to  the  unexpectedly  long  time  frame  for  review, 
although  other  reasons  also  became  apparent.  To  achieve  the  study  goals,  several 
modifications  were  necessary.  Through  discussions  with  the  research  associates, 
the  investigators  refined  the  process,  decreasing  the  time  required  for  data  entry, 
while  increasing  time  for  meaningful  analysis. 

The  first  set  of  refinements  streamlined  the  overall  methods.  Because 
discrepancies  between  the  CMR  and  the  pharmacy  computer  system  were 
ubiquitous,  the  substudy  was  eliminated.  Because  comprehensive  review  of  each 
medication  regimen  was  burdensome,  review  was  narrowed  to  focus  on  data 
relevant  solely  to  the  prescription  undergoing  evaluation.  The  number  of  months 
included  in  the  retrospective  review  of  the  CMR  was  decreased  from  15  to  6, 
unless  the  specific  instance  required  additional  evaluation,  at  the  discretion  of  the 
research  associate.  Double  entry  of  data  was  eliminated,  resulting  in  each 
prescription  being  reviewed  once  by  one  research  associate.  This  refinement  will 
be  accounted  for  in  the  statistical  analysis,  by  conducting  a  test  of  interrater 
reliability. 

The  next  set  of  refinements  involved  revision  of  several  aspects  of  the  data 
entry  process.  Although  the  numerous  background  tables  were  intended  to  create 
efficiencies  during  the  data  entry  process,  only  some  tables  did,  and  others  did 
not.  The  background  tables  presented  too  may  choices  from  which  to  choose.  The 
allergy  table  was  comprised  of  more  than  400  possibilities;  the  “directions  for 
use”  (“sig,”  signe’tour)  table  was  comprised  of  more  than  1 1,000  “sigs,”  and  the 
table  of  ICD-9-CM  codes  was  comprised  of  more  than  15,000  records.  Further, 
the  background  tables  that  linked  drugs  to  dosage  forms  and  doses  were  too 
restrictive.  The  research  associates  were  frequently  unable  to  enter  the  letters  of 
the  alphabet  (or  numbers)  that  called  up  the  data  that  would  match  exactly,  the 
prescription  they  were  evaluating.  Consequently,  the  database  programmer 
created  two  shortcuts:  an  “express  sig”  list  of  20  common  “sigs,”  and  a 
consolidated  ICD-9-CM  list  comprised  of  58  composite  disease  states  (sorted 
alphabetically).  Hard  copies  of  each  of  these  lists  were  added  to  the  Reference 
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Packets  at  each  computer  terminal.  The  programmer  also  unlinked  drugs  from 
dosage  forms  and  doses,  allowing  the  research  associates  to  enter  each  of  these 
elements  separately.  As  a  final  feature,  free-text  fields  were  added  in  strategic 
places  to  facilitate  the  entry  of  data  without  restrictions.  Once  added,  these  free- 
text  entries  were  saved  by  the  system  and  could  be  recalled  for  future  use.  A  free- 
text  field  was  also  added  for  comments. 

Two  of  the  investigators  conducted  quality  audits  of  10  percent  of  the 
prescriptions  entered.  These  reviews  pointed  up  the  need  for  additional  decision 
rules  about  what  constituted  a  medication  error.  These  were  addressed  in  a 
systematic  fashion,  codified  in  the  Operations  Manual,  and  added  to  the 
Reference  Packet.  The  team  created  a  decision  rule  that  required  a  distinction  be 
made  between  tablets  and  capsules  (for  purposes  of  identifying  an  error)  only 
when  the  dosage  form  impacted  the  phannacokinetics  or  pharmacodynamics  of 
the  drug  (e.g.,  an  immediate-  versus  sustained-release  formulation).  Separately, 
the  team  addressed  the  issue  of  whether  the  dispensing  of  a  drug  with  a  particular 
strength  and  directions  different  from  those  prescribed  constituted  an  error,  for 
example,  dispensing  “atorvastatin  20  mg,  Vi  tablet  daily”  for  a  prescription  written 
for  “atorvastatin  10  mg  once  daily.”  As  this  is  considered  standard  practice  in  the 
managed  care  setting,  this  was  codified  as  not  an  error. 

With  close  attention  to  details  and  revisions  such  as  these,  the  time  for 
evaluation  of  each  prescription  decreased  to  approximately  5-6  minutes  for  a 
minimally  complex  patient,  and  7-10  minutes  for  a  moderately  complex  patient. 
Even  so,  the  evaluation  of  1,500  prescriptions  required  more  than  400  hours  of 
labor.  The  evaluation  was  completed  in  May  2003. 

Current  status  of  the  implementation 

This  pilot,  pre-ACPOE  data,  describing  the  incidence  and  characterizing  the 
epidemiology  of  medication  errors  using  the  handwritten  prescribing  process,  are 
now  being  analyzed.  These  results  will  be  published  in  a  subsequent  paper. 
Implementation  of  the  ACPOE  system  in  the  pilot  clinic  took  place  during  July 
2003.  The  prescribers  have  had  12  months  to  become  proficient  with  the  use  of 
the  new  system.  The  collection  of  the  postimplementation  phase  of  the  study  is 
imminent.  In  the  meantime,  the  ACPOE  system  is  being  rolled  out  to  additional 
clinics  within  the  Everett  Clinic  integrated  health  system. 


Discussion 

This  report  describes  our  experience  in  conducting  a  small  pilot  study  that 
evaluated  medication  errors  prior  to  implementation  of  an  ACPOE  system. 
Although  it  is  standard  to  include  a  “methods”  section  in  the  publication  of  the 
results  of  any  similar  study,  we  have  not  found  a  paper  in  this  field  that  has 
focused  solely  on  providing  the  details  of  study  design  and  data  collection 
methods.  Our  goal  in  so  doing  is  to  convey  to  others  the  complexity  of  even  such 
a  seemingly  simple  evaluation  process. 
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With  implementation  of  ACPOE  systems  becoming  more  widespread  in  the 
coming  years,  this  experience  may  be  useful  to  others  as  they  undertake  similar 
evaluations.  Although  aspects  of  our  evaluation  are  unique  to  the  Everett  Clinic 
setting,  others  are  generalizable.  This  study  was  conducted  in  the  primary  care 
setting  (internal  medicine)  of  a  community-based  integrated  health  system.  The 
types  of  errors  that  are  characterized  in  the  ambulatory  setting  are  often  different 
from  those  occurring  in  the  inpatient  environment,  but  perhaps  similar  among 
different  ambulatory  settings.  Our  taxonomy  for  characterizing  errors  may  be 
useful  in  creating  a  standard  taxonomy  for  the  ambulatory  setting.  Our  approach 
to  evaluating  medication  errors  associated  with  laboratory  monitoring  may  inform 
decision  support  programming  in  the  future. 

The  study  design  enabled  the  capture  of  infonnation  on  severity  levels  for 
patients  who  were  harmed  by  an  error  and  subsequently  seen  in  the  clinic  setting. 
It  did  not  include  those  for  whom  a  hospital  admission  or  emergency  room  visit 
was  necessary.  Because  the  design  did  not  include  an  evaluation  of  medications 
dispensed,  the  results  will  be  limited  in  distinguishing  between  severity  levels  “B” 
and  “C”  on  the  NCC  MERP  Risk  Assessment  Index  (Table  2).  The  investigators 
will  further  explore  these  limitations  during  data  analysis. 

Several  commonly  held  beliefs  about  prescribing  were  noted  in  this  study. 
Prescribers  seldom  write  with  the  same  degree  of  detail  and  the  same  level  of 
accuracy,  as  required  in  the  drug  dispensing  process.  Definitions  of  medication 
errors  differ,  depending  on  one’s  perspective.  A  prescription  considered  error-free 
by  a  prescriber  may  be  considered  by  a  phannacist  to  contain  an  error;  a 
prescription  considered  error-free  by  a  pharmacist  may  still  cause  a  patient  to  be 
confused,  thus  qualifying  to  be  an  error.  Patient  information  contained  in  the 
medical  record  is  seldom  consistent  with  that  contained  in  the  pharmacy  computer 
system — even  in  an  integrated  health  system  with  a  systemwide  CMR. 


Conclusion 

By  paying  close  attention  to  detail  during  the  design  of  the  study,  addressing 
each  challenge  as  it  arose  and  streamlining  methods,  the  research  team 
implemented  a  process  to  evaluate  prescriptions  for  medication  errors.  This 
method  will  next  be  used  to  identify  and  characterize  medication  errors  in  the 
postimplementation  phase  of  ACPOE  implementation. 

The  potential  for  improvements  in  our  nation’s  health  care  system  offered  by 
IT,  and  specifically  by  ACPOE,  is  substantial.  Even  using  IT  to  evaluate 
medication  errors  requires  careful  planning.  Realizing  the  benefits  of  ACPOE 
systems  will  require  the  same  level  of  careful  planning  and  attention  to  detail.  A 
carefully  crafted  methodology,  implementation  plan,  and  response  to  challenges 
as  they  arise  are  critical  to  the  success  of  any  IT-related  patient  safety  endeavor. 
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